Method of presenting a piece of music to a user of an electronic device

ABSTRACT

A method of presenting music to a user of an electronic device comprises the step of providing score data representing the musical score of the piece of music in a graphical representation, audio data representing a recording of the piece of music, and music data representing one or more parts of parts of the piece of music in a digital format such as MIDI or MusicXML. The music data representing a part of the music that has been selected by the user is transformed into part sound signals using a sound generator. The part sound signals and audio sound signals are merged so as to obtain a merged sound signal in which the piece of music as represented by the music data file and by the audio data file are synchronized. Finally, and simultaneously, the sound of the piece of music using the merged sound signal is played audibly, the musical score is displayed on a display, and a sub-portion of the musical score corresponding to a passage of the piece of music which is presently audible is highlighted on the display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of European patent application14003079.2 filed Sep. 5, 2014. The full disclosure of this earlierapplication is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to the field of tools andmethods for presenting music. The disclosure particularly pertains totablet computers or other electronic devices used for digital sheetmusic display and music audio playback.

2. Description of Related Art

A couple of years ago sheet music publishers began to publish music alsoin paperless digital formats. With the release of tablet appliances suchas the iPad®, many different digital software applications (hereinafter“apps”) for displaying sheet music have entered the market. The first ofthese apps displayed digital sheet music as stored in Portable DocumentFormat (PDF) files. Soon afterwards these apps added the ability toannotate the sheet music with graphical markings. More recent apps addedthe capability, for use with suitable hardware platforms, to play anaudio file simultaneously with displaying the sheet music.

Still later generations of tablet apps added more interactivity. Forexample, the NoteStar® app from Yamaha allows musicians to listen tomusic playing back in synchrony with a scrolling digital sheet musicdisplay.

So far all commercially available sheet music apps represent sheet musiceither in native app formats or standard cross-application formats likeMusicXML and PDF files.

MusicXML files currently have no standard mechanism for synchronizingaudio files to notation files for synchronized playback. Furthermore,MusicXML files represent music in a logical format structured aroundMIDI playback, rather than a visual format structured around display.This complicates the process of displaying sheet music on an electronicdevice in a form that captures the fine engraving details of a paperpublication. In other words, the quality of the sheet music displayed onthe electronic device is often poor if compared with a good paperpublication of the same piece of music. For producing such paperpublications huge efforts are often invested by the publishers toarrange the notes and other symbols in a manner that can be easilygrasped by performers, students or music fans. This particularly appliesto orchestral or other polyphonic music in which a plurality of partsare arranged one below the other on a single sheet to form a musicalscore.

The PDF format, on the other hand, is capable of representing sheetmusic in a high quality format. But since it does not contain, incontrast to the MusicXML format, any information that relates to themusic as such, it has not been used to play an audio or video filesimultaneously with displaying the sheet music.

For these reasons the sheet music apps currently available sacrifice, atleast to some extent, the high layout quality of printed sheet musicalready available to the ability to play an audio or video filesimultaneously with displaying the sheet music.

U.S. Pat. No. 7,239,320 B1 discloses a computer implemented method ofpresenting music to a user of an electronic device. This method makes itpossible to display sheet music in a high quality graphicalrepresentation and to play an audio or video file simultaneously. Tothis end a hierarchical structure of bounding boxes is used tosynchronize displayed sheet music to a musical performance recorded asaudio or video file.

However, although this method is very helpful to compare a musicalperformance with corresponding sheet music, it cannot provide much helpto musicians or students who want to improve their skills in performinga certain part in that piece of music. For example, if a chorister wantsto better learn his part with the help of a good audio recording, it isusually difficult to clearly identify this part in the recording.

Prior art approaches use for such educational purposes a MIDIrepresentation of the piece of music. Then it is easy to highlight acertain part by simply increasing the volume of the MIDI channel thatcorresponds to this particular part. But music produced on the basis ofa MIDI file is often not satisfactory for a listener, because it lacksthe charm and natural atmosphere of a recorded performance of realmusicians. Amplifying a specific part in an audio recording is notpossible because the audio recording is obtained by sampling an analogwave signal from which no specific parts can be isolated.

SUMMARY OF THE INVENTION

It is therefore an object of the invention to provide a method ofpresenting music to a user of an electronic device which may be helpfulto him to learn a part of a piece of music.

It is a further object of the invention to provide such a method whichsupports a musician or student to identify more clearly a particularpart if listening to a recording of a polyphonic piece of music.

These and other objects are achieved with a method in accordance withthe present invention. The method comprises the step of providing atleast following three data files to an electronic device. A score datafile contains score data representing the musical score of the piece ofmusic in a graphical representation, wherein the musical score comprisesa plurality of parts. An audio data file contains audio datarepresenting the piece of music as played by musicians in a format inwhich sampled analog sound signals are encoded. A music data filecontains music data representing at least one of the plurality of partsof the piece of music in a digital format in which pitch and relativeduration of sounds are encoded.

A user is allowed to select one of the parts that shall be acousticallyhighlighted when audibly playing the piece of music. The music data,which represent the part that has been selected by the user in step b),is then transformed into part sound signals using a sound generator.Audio sound signals are produced on the basis of the audio data. Thepart sound signals and the audio sound signals are merged so as toobtain a merged sound signal in which the piece of music as representedby the music data file and the piece of music as represented by theaudio data file are synchronized. Finally, and simultaneously, the soundof the piece of music using the merged sound signal is played audibly,at least a portion of the musical score is displayed on a display, and asubportion of the musical score that corresponds to a passage of thepiece of music which is presently audible is highlighted on the display.

BRIEF DESCRIPTION OF THE DRAWINGS

Various features and advantages of the present invention may be morereadily understood with reference to the following detailed descriptiontaken in conjunction with the accompanying drawing in which:

FIG. 1 is top view on a tablet computer;

FIG. 2 is a flow diagram illustrating important steps that are carriedout by the computer shown in FIG. 1;

FIG. 3 is a page of a musical score in a graphical representation;

FIG. 4 is a graph illustrating an audio sound signal;

FIG. 5 is an excerpt from a MidiXML file;

FIG. 6 is a graph illustrating the synchronization of the music datafile and the audio data file.

The subject matter given herein includes references to commercialproducts offered under product names for which trademark rights mayapply. Where applicable, such trademark rights are the property of theirowners of record.

DESCRIPTION OF PREFERRED EMBODIMENTS

1. General Function of Tablet Computer

FIG. 1 is a schematic top view on a tablet computer 10 having a memoryin which an app is stored that will be explained further below in moredetail. The term app is used in this context synonymously for all kindsof computer programs irrespective of the programming language or thecomputer platform.

The tablet computer 10 as such is a conventional electronic handhelddevice that is provided, in this particular embodiment, with a touchsensitive display 12, loudspeakers 14, 16 and control buttons 18, 20. Asa matter of course, also other types of computers may be used instead.The term computer is to be construed broadly and comprises mobilephones, notebooks, PCs or workstations, but also electronic devices witha specific function such as a keyboard in which the app is stored in aread-only memory.

In FIG. 1 it can be seen that a musical score of a piece of music, herethe first page of a piano score 22 of the “Mass in C major” from L. v.Beethoven, is shown on the display 12. The piano score 22 comprises fourstaffs for the different voice parts and two staffs for the piano part.In this embodiment the page shown on the display 12 is an identical copyof a paper edition published by the applicant, and thus it has the sameperfect layout as the paper edition. Thus a user of the tablet computer10 who is already familiar with the paper edition, for example becausehe or she is sometimes singing in a choir, can easily track the music onthe display. It may also be possible to show only a portion of this pageon the display 12. One reason for doing this might be that the display12 is too small to show the entire page.

While the first page of the piano score 22 is displayed, a professionalrecording of this piece of music is played back using the loudspeakers14, 16. This recording may be obtained from a CD by compressing theaudio data, for example, using the MP3 standard. The portion of thescore 22 which is presently audible is highlighted on the display withthe help of a grey or colored bar 24 which moves, synchronized with themusic presently played, over the musical score 22.

If a user is interested in a particular part of this piece of music,e.g. the bass part of the choir, he may simply touch with one of hisfingers the display 12 at a position where the bass staff is shown. Thiswill be interpreted by the tablet computer 10 as a command to highlightthe bass part while the music is still playing.

The tablet computer will now play the music, but with acousticallyhighlighting the part selected by the user. The user will hear not onlythe recording of the piece of music, but additionally the selected partin a MIDI representation that has been produced by a sequencer oranother sound generator.

How this is accomplished will be explained in the following withreference to FIG. 2 which is a flow diagram illustrating important stepsthat are carried out by the tablet computer 10.

2. Computer Program

In a first step S1, the tablet computer 10 is provided with threedifferent data files. These data files may be downloaded from a serveror may be provided to the tablet computer using a data carrier such as aUSB stick, for example.

A first data file is a score data file containing score data thatrepresent the musical score, which is in this particular embodiment apiano score, of the piece of music in a graphical representation as itis shown in FIG. 3. The term “graphical representation” denotes a formatin which the score is encoded as graphical information and not as logicinformation. The score data file may be, for example, a TIFF (TaggedImage File Format) file that is used for storing raster graphics images,or a PDF file in which raster graphics images are embedded afterfiltering. It is also possible to embed the notes and other symbols ofthe musical score similar to embedded fonts.

A second data file is an audio data file containing audio datarepresenting the piece of music as played by musicians in a format inwhich sampled analog sound signals are encoded. FIG. 4 shows an analogsound signal before it is sampled and encoded. It can be seen that inthis analog signal it is not possible to identify the bass voice or andany other specific part.

The audio file may be an uncompressed data file such as a file in theWaveform Audio File Format (WAVE) or the Compact Disc Digital AudioFormat (CDDA). Since the data storage capacity of the tablet computer 10is restricted, it may be advantageous to use a compressed audio datafile, for example a file in the MP3 format, the Advanced Audio Codingformat (AAC), the Free Lossless Audio Codec format (FLAC) and theOGG/Vorbis format.

The third file provided in step S1 is a music data file containing musicdata that represent, in the embodiment shown, individually the fourvoice parts of the piece of music in a digital format in which pitch andrelative duration of sounds are encoded. The most common file format ofthis kind is the MIDI format. MIDI, which is an acronym for MusicalInstrument Digital Interface, is a technical standard for a protocolthat allows a wide variety of electronic musical instruments, computersand other related devices to connect and communicate with one another.

MIDI carries event messages that specify notation, pitch and velocity,control signals for parameters such as volume, vibrato, audio panning,cues, and clock signals that set and synchronize tempo between multipledevices. These messages are sent to other devices where they controlsound generation and other features. These data can also be recordedinto a hardware or software device called a sequencer, which can be usedto edit the data and to play them back at a later time.

Since MIDI files are stored as a sequence of hexadecimal numerals, itmay be more convenient to use the MidiXML format for the music data fileinstead. FIG. 5 is an excerpt from a MidiXML file representing the firstnote of Beethoven's piece of music shown in FIG. 3. These data may betransformed into a MIDI file, which may then be supplied to a sequencerto generate an analog sound signal. It is also possible to produce, onthe basis of MidiXML data, a musical score using a notation program suchas FINALE® that translates the MidiXML data into a graphicalrepresentation.

Here it is assumed that the music data file is a MidiXML file in whichat least the pitch and the duration of the notes in the four voice partsof the piece of music are encoded in a manner that is illustrated inFIG. 5.

In a second step S2 the user is allowed to select one of the four voiceparts that shall be acoustically highlighted when the piece of music isaudibly played. As mentioned further above, this selection may be madeby touching the staff on the display that corresponds to the selectedvoice part. Alternatively, the user may be able to make this selectionin a menu that pops up if he presses one of the buttons 14, 16. Stillfurther, the user may select his personal voice once at the beginning ofthe program. While the music is audibly played, an additional buttonlabeled “Emphasize my voice”, for example, appears on the display 12.The selection is then made by pressing this button. This approach hasthe advantage that the user does not have to identify his particularvoice each time he wants to use the highlighting function.

In a third step S3 the music data, which represent the voice part thathas been selected by the user in step S2, is transformed into part soundsignals using a sound generator. To this end the tablet computer 10transforms the MidiXML data of the music data file into MIDI data. Thesound generator, which may be realized as a MIDI sequencer programmodule, transforms the MIDI data into an analog sound signal. The soundproduced by the sound generator may be adapted to the part that shall beacoustically highlighted. Thus, if the user desires the bass voice to beacoustically highlighted, the sound generator may produce an artificialsound that resembles a human bass voice. Since—particularly in the caseof human voices—such an artificial sound is sometimes unsatisfactory, aneutral sound, for example a piano sound, may be produced instead by thesound generator using the MIDI data. In an alternative embodiment, thetablet computer 10 is provided in step S1 not with a MidiXML file, butwith a MIDI file. Then no MidiXML data have to be translated into MIDIdata by the tablet computer 10.

In a fourth step S4 audio sound signals are produced by the computer 10on the basis of the audio data contained in the audio data file. Thismay be accomplished with a conventional decoder that is capable ofdecoding the digital audio data so as to produce an analog audio signalas shown in FIG. 4.

In a next step S5 the part sound signals produced by the sound generatorand the audio sound signals produced by the decoder are merged so as toobtain a merged sound signal in which the piece of music as representedby the music data file and the piece of music as represented by theaudio data file are synchronized. The merging may be performed at adigital level or alternatively at an analog level.

3. Synchronization

A first approach for synchronizing the piece of music represented by themusic data file and the audio data file will be described in thefollowing with reference to Table 1.

TABLE 1 MidiXML data Audio recording Δ Original Δ Modified # beat time t[s] # Ticks Tempo [μs] Tempo [μs] 0 0 0 1360000 1361000 1 1.361 2561360000 1395000 2 2.756 512 1360000 1241000 3 3.997 768 1360000 13220004 5.319 1024 1360000 1361000 . . . . . . . . . . . .

The left column in Table 1 represents the beat number in one of the barsof the piece of music. The second column indicates the time in secondsat which the beats occur in the audio recording. This information may beobtained “manually” by a person who listens to the recording and pressesa button each time a beat is felt. It can be seen that although onemight expect that the beats occur at strictly periodic intervals, theydo not in the audio recording, perhaps because the conductor decided tomodify the tempo between the beats.

The third column denotes the corresponding information in the originalMidiXML file. Each beat is divided into 256 ticks, and the tempoinformation for each beat is 136000 μs. In contrast to the audiorecording, the time sequence is described here by the time interval Δbetween two consecutive beats.

In the original MidiXML data this time interval Δ is 1360000 μs=1.36 sfor each beat. Therefore, if one would transform the MidiXML file into aMIDI file, produce an audio sound and play it together with the audiorecording, the two sounds would not match, because the audio soundobtained from the MIDI data would be perfectly regular, whereas theaudio recording contains tempo modifications.

Therefore the MidiXML data relating to the time interval Δ is modifiedsuch that the two sounds are synchronized. To this end the modified timeinterval data Δ for beat n is changed such that it is equal to thedifference d=t_(n)−t_(n-1). In this manner the duration Δ of soundsencoded in the digital music file is determined such that the piece ofmusic as represented by the music data file and the piece of music asrepresented by the audio data file are synchronous when simultaneouslyplayed.

FIG. 6 shows a graph that illustrates this way of synchronizing thetimeline of the MidiXML data file with the timeline of the audio datafile. In the original MidiXML data file the beats occur strictlyperiodically, whereas in the audio recording the beats occur somewhatirregularly. By modifying the time intervals between the beats in theMidiXML file, it is possible to match the data such that music producedon the basis of the music data file and music produced on the basis ofthe audio file is synchronized.

Another way of synchronizing the audio data file with the music datafile is to provide a separate synchronization file to the tabletcomputer 10. This synchronization file contains synchronization dataestablishing a time correlation between the music data and the audiodata. These synchronization data are used by the sound generator when ittransforms the music data into part sound signals in step S3.

4. Synchronized Playback and Part Highlighting

In a final step S6 the sound of the piece of music is audibly played viathe loudspeakers 14, 16 using the merged sound signal. Simultaneously atleast a portion of the piano score 22 is displayed on the display 12,and a portion of the musical score that corresponds to a passage of thepiece of music which is presently audible is highlighted using the bar24.

5. Volume Control

The volume of the sound that is produced by the sound generator on thebasis of the music data file is originally not adapted to the volume ofthe sound produced by the decoder on the basis of the audio data. Thus,if the music of the recording becomes very loud at certain passages, itmight be impossible to discern the additional sound that is merged intothe recording for highlighting a certain part of the music.

Therefore it may be considered to appropriately modify sound volumedata, which are contained in the music data, at the time the part soundsignals and the audio sound signals are merged in step S5. Usually itwill be desirable to increase the volume of the part sound signals withincreasing volume of the audio sound signals, and vice versa. But alsomore complicated and non-linear schemes may be applied during themodification of the volume data.

The above description of the preferred embodiments has been given by wayof example. From the disclosure given, those skilled in the art will notonly understand the present invention and its attendant advantages, butwill also find apparent various changes and modifications to thestructures and methods disclosed. The applicant seeks, therefore, tocover all such changes and modifications as fall within the spirit andscope of the invention, as defined by the appended claims, andequivalents thereof.

The invention claimed is:
 1. A method of presenting a piece of music toa user of an electronic device, said method comprising the followingsteps: a) providing the following data files to the electronic device: ascore data file containing score data representing the musical score ofthe piece of music in a graphical representation, wherein the musicalscore comprises a plurality of parts, an audio data file containingaudio data representing the piece of music as played by musicians in aformat in which sampled analog sound signals are encoded, a music datafile containing music data representing at least one of the plurality ofparts of the piece of music in a digital format in which pitch andduration of sounds are encoded, b) allowing a user to select one of theparts that shall be acoustically highlighted when audibly playing thepiece of music; c)transforming the music data, which represent the partthat has been selected by the user in step b), into part sound signalsusing a sound generator; d) producing audio sound signals on the basisof the audio data; e) merging the part sound signals and the audio soundsignals so as to obtain a merged sound signal in which the piece ofmusic as represented by the music data file and the piece of music asrepresented by the audio data file are synchronized; f) simultaneouslyaudibly playing the sound of the piece of music using the merged soundsignal, displaying at least a portion of the musical score on a displayand highlighting on the display a sub-portion of the musical score thatcorresponds to a passage of the piece of music which is presentlyaudible.
 2. The method of claim 1, wherein the digital format is a MIDIformat and the music data file is a MIDI file.
 3. The method of claim 1,wherein the digital format is a MusicXML format and the music data fileis a MusicXML file.
 4. The method of claim 1, wherein in the digitalformat an absolute duration of sounds is encoded, and wherein theduration of sounds is determined such that the piece of music asrepresented by the music data file and the piece of music as representedb the audio data file are synchronous when simultaneously played in stepf).
 5. The method of claim 1, wherein a synchronization file is providedto the electronic device in step a), said synchronization filecontaining synchronization data establishing a time correlation betweenthe music data and the audio data, and wherein the synchronization dataare used by the sound generator when transforming the music data intopart sound signals in step c).
 6. The method of claim 1, wherein theaudio data file is an uncompressed data file provided in a formatselected from the group consisting of: Waveform Audio File Format (WAVE)and Compact Disc Digital Audio Format (CDDA).
 7. The method of claim 1,wherein the audio data file is a compressed data file provided in aformat selected from the group consisting of: MP3 format, Advanced AudioCoding format (AAC). Free Lossless Audio Codec format (FLAC) and Theora.8. The method of claim 1, wherein the music data comprise sound volumedata, and wherein, when merging the part sound signals and the audiosound signals in step e), the sound volume data are modified dependingon a volume of the audio sound signals.
 9. The method of claim 8,wherein the volume of the part sound signals is increased withincreasing volume of the audio sound signals.
 10. The method of claim 1,wherein the audio data file is contained in a multimedia container filethat contains also a video data file.
 11. A non-transitory computerreadable memory with a computer program stored thereon, the computerprogram configured to cause a computer to perform the following steps:a) loading the following data files to the computer: a score data filecontaining score data representing the musical score of the piece ofmusic in a graphical representation, wherein the musical score comprisesa plurality of parts, an audio data file containing audio datarepresenting the piece of music as played by musicians in a format inwhich sampled analog sound signals are encoded, a music data filecontaining music data representing at least one of the plurality ofparts of the piece of music in a digital format in which pitch andduration of sounds are encoded, b) allowing a user to select one of theparts that shall be acoustically highlighted when audibly playing thepiece of music; c) transforming the music data, which represent the partthat has been selected by the user in step b), into part sound signals;d) producing audio sound signals on the basis of the audio data; e)merging the part sound signals and the audio sound signals so as toobtain a merged sound signal in which the piece of music as representedby the music data file and the piece of music as represented by theaudio data file are synchronized; f) simultaneously audibly playing thesound of the piece of music using the merged sound signal using aloudspeaker or earphones, displaying at least a portion of the musicalscore on a display and highlighting on the display a sub-portion of themusical score that corresponds to a passage of the piece of music whichis presently audible.
 12. The computer readable memory of claim 11,wherein the digital format is a MIDI format and the music data file is aMIDI file.
 13. The computer readable memory of claim 11, wherein thedigital format is a MusicXML format and the music data file is aMusicXML file.
 14. The computer readable memory of claim 11, wherein inthe digital format an absolute duration of sounds is encoded, andwherein the duration of sounds is determined such that the piece ofmusic as represented by the music data file and the piece of music asrepresented by the audio data file are synchronous when simultaneouslyplayed in step f).
 15. The computer readable memory of claim 11, whereina synchronization file is provided to the electronic device in step a),said synchronization file containing synchronization data establishing atime correlation between the music data and the audio data, and whereinthe synchronization data are used by the sound generator whentransforming the music data into part sound signals in step c).
 16. Thecomputer readable memory of claim 11, wherein the audio data file is anuncompressed data file provided in a format selected from the groupconsisting of: Waveform Audio File Format (WAVE) and Compact DiscDigital Audio Format (CDDA).
 17. The computer readable memory of claim11, wherein the audio data file is a compressed data file provided in aformat selected from the group consisting of: MP3 format, Advanced AudioCoding format (AAC), Free Lossless Audio Codec format (FLAC) and Theora.18. The computer readable memory of claim 11, wherein the music datacomprise sound volume data, and wherein, when merging the part soundsignals and the audio sound signals in step e), the sound volume dataare modified depending on a volume of the audio sound signals.
 19. Thecomputer readable memory of claim 18, wherein the volume of the partsound signals is increased with increasing volume of the audio soundsignals.
 20. The computer readable memory of claim 11, wherein the audiodata file is contained in a multimedia container file that contains alsoa video data file.
 21. The computer readable memory of claim 11, whereinthe computer is a tablet computer.
 22. The computer readable of claim11, wherein the computer readable memory is a part of the computer. 23.A method of supporting a user to learn a piece of music, comprising thefollowing steps: a) displaying at least a portion of a musical score ofthe piece of music on a display, wherein the musical score comprises aplurality of parts; b) allowing a user to select one of the parts thatshall he acoustically highlighted when audibly playing the piece ofmusic; c) producing audio sound signals on the basis of audio datarepresenting the piece of music as played by musicians in a format inwhich sampled analog sound signals are encoded; d) producing part soundsignals using a sound generator and music data which represent the partthat has been selected by the user in step b); e) merging the part soundsignals and the audio sound signals so as to obtain a merged soundsignal in which the piece of music as represented by the music data fileand the piece of music as represented by the audio data file aresynchronized; f) audibly playing the sound of the piece of music usingthe merged sound signal; g) highlighting on the display a sub-portion ofthe musical score that corresponds to a passage of the piece of musicwhich is presently audible.